{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 5.0 MPI-Parallelization with NGSolve\n",
    "\n",
    "\n",
    "### This tutorial was prepared by Lukas Kogler for 2018 NGSolve-usermeeting. It was not updated since then, and some parts may be outdated. Nevertheless, it might be a source of inspiration\n",
    "\n",
    "This talk will be split into two parts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "In part one of the talk, we will look at the basics:\n",
    "\n",
    "* How do we start a distributed computation\n",
    "    * mpirun/mpiexec\n",
    "    * batch systems\n",
    "    * jupyter-magic\n",
    "* Basics of MPI-parallel NGSolve\n",
    "    * distrubuted meshes\n",
    "    * distributed Finite Element Spaces\n",
    "    * distributed Linear Algebra\n",
    "    * solvers & preconditioners\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Part two will be focussed on the FETI-DP method and it's implementation in NGSolve\n",
    "an will be in collaboration with Stephan Köhler from TU Bergakademie Freiberg. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## MPI - Message Passing Interface\n",
    "\n",
    "With an MPI-library, multiple seperate processes can exchange data very easily\n",
    "and thus work together to do large computations.\n",
    "\n",
    "Within an MPI job, each process in a computation is given a \"rank\", a numer from $0\\ldots n_p$,\n",
    "which is used as it's identifier.\n",
    "\n",
    "Communication happens within so-called 'mpi-communicators', which are contexts within which\n",
    "messages can be exchanged."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Running computations with MPI \n",
    "\n",
    "If you are familiar with MPI, you already know the dos and don'ts, and if you are following the presentation\n",
    "on your own machine I cannot tell you what to do.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Directly - mpiexec\n",
    "In the simplest case, we can start an MPI program with  **mpiexec -np N some_program**.\n",
    "\n",
    "In our case, we want to start N instances of python\n",
    "**mpiexec -np N ngspy my_awesome_computation.py**\n",
    "\n",
    "On clusters, however, this is usually not an option.\n",
    "\n",
    "\n",
    "### We ask you not to do this if you use the cluster (it will run the computation on the login node!)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Jupyter-Magic\n",
    "For the purposes of this presentation, we have set up jupyter-notebooks on the COEUS cluster at\n",
    "Portland State University.\n",
    "\n",
    "We thank PICS, the Portland Institute for Computational Science for granting us access\n",
    "and organizing user accounts.\n",
    "\n",
    "We would also like to acknowledge NSF Grant# DMS-1624776 which gave the funding for \n",
    "the cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To connect, follow the following steps:\n",
    "\n",
    "You should have gotten an email with 2 attached files:\n",
    "\n",
    "     connectme.py  usrmtg_user_data\n",
    "\n",
    "Download those and call\n",
    "     \n",
    "     python3 connectme.py your_lastname\n",
    "     \n",
    "Follow the instructions, and you will be connected to your own jupyter-notebook\n",
    "running on COEUS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### How to use it\n",
    "We can start a \"cluster\" of python-processes. The cluster will be identified by some \"user_id\".\n",
    "While it is running, it will allocate N cores (in this case 5), to this specific cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for connection file: ~/.ipython/profile_ngsolve/security/ipcontroller-kogler-client.json\n",
      "connecting ... try:6 succeeded!"
     ]
    }
   ],
   "source": [
    "num_procs = '5'\n",
    "from usrmeeting_jupyterstuff import start_cluster, connect_cluster\n",
    "start_cluster(num_procs)\n",
    "connect_cluster()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Python code in a normal cell will be excecuted as usual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this is executed right here\n"
     ]
    }
   ],
   "source": [
    "print('this is executed right here')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Python code in a cell with that has %%px in the first line will be executed by all workers in the cluster in parallel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[stdout:0] hello from everyone in the cluster!\n",
      "[stdout:1] hello from everyone in the cluster!\n",
      "[stdout:2] hello from everyone in the cluster!\n",
      "[stdout:3] hello from everyone in the cluster!\n",
      "[stdout:4] hello from everyone in the cluster!\n"
     ]
    }
   ],
   "source": [
    "%%px\n",
    "print('hello from everyone in the cluster!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[stdout:3] hello from some in the cluster\n",
      "[stdout:4] hello from some in the cluster\n"
     ]
    }
   ],
   "source": [
    "%%px --targets 3:5\n",
    "print('hello from some in the cluster')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can shut down the cluster again. **This frees the resources allocated for the cluster!!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from usrmeeting_jupyterstuff import stop_cluster\n",
    "stop_cluster()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "## Please clean up your clusters!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Batch System\n",
    "On clusters, we usually have to make use of a batch system\n",
    "The details depend on the specific system.\n",
    "\n",
    "COEUS uses SLURM (**S**imple **L**inux **U**tility for **R**esource **M**anagement), and \n",
    "we have prepared ready to go job submission scripts. \n",
    "\n",
    "For each *file.ipynb*, there is a file *file.py* and a slurm-script *slurm_file*, which can be submitted \n",
    "with the command\n",
    "\n",
    "** sbatch slurm_file **\n",
    "\n",
    "You can check the status of your jobs with **squeue -u username**.\n",
    "\n",
    "The slurm-scripts can be opened and modified with a text editor if you want to experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The Rest of the talk:\n",
    "For introductory part I will be using:\n",
    "\n",
    "- [Basics](../unit-5.1-mpi_ngsolve/mpi_basics.ipynb): Distributed Meshes, Finite Element Spcaces and Lienar Algebra\n",
    " \n",
    "For the FETI-DP part, the files are:\n",
    "\n",
    "- [FETI-DP I](../unit-5.2-fetidp_point2d/feti-dp-i.ipynb): Point Constraints in 2d\n",
    "- [FETI-DP II](../unit-5.3-fetidp_point3d/feti-dp-ii.ipynb): Point Constraints in 3d\n",
    "- [FETI-DP III](../unit-5.4-fetidp_edge/feti-dp-iii.ipynb): Point- and Edgeconstraints\n",
    "- [FETI-DP IV](../unit-5.5-fetidp_inexact/feti-dp-iv.ipynb): Inexact FETI-DP\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
